The disclosure relates generally to verification for designs, and, more particularly, to rank fault-test pairs based on waveform statistics in a mutation-based test program evaluation system.
It is well known in integrated circuit (IC) design that a complex integrated circuit can contain errors. Different techniques are therefore used to check the operation of an integrated circuit or of a computer program modeling the integrated circuit. Typically, a test suite is performed, each test indicating whether an error has been detected or not in the integrated circuit.
Mutation analysis is a test evaluation technique which requires large computing resources. When mutation analysis is performed, errors, faults, or mutations are inserted in an initially non-mutated program or design code. Then, a check is made to determine whether the tests can detect the inserted mutations. When, with a predetermined test, a mutated program provides an identical result to a result provided by the non-mutated program executed with the predetermined test, the mutation is identified as being non-detected. On the other hand, when the result provided by the mutated program, with a predetermined test, is different from the result provided by the non-mutated program executed with the predetermined test, the mutation is identified as being detected. The test is stopped and a next mutation analysis is performed. The tests enabling mutations to be detected are retained for an improved test suite. The advantage of mutation analysis is to evaluate the quality of the test suite and to optimize the test suite if necessary.
If the tests are performed for a large number of mutations, mutation analysis may take a very long time. In this case, the number of mutations may be reduced, for example by choosing a certain kind of mutations or by randomly choosing a certain number of mutations.
U.S. Pat. No. 7,574,681 discusses a mutation-based test program evaluation system that obtains the waveforms from a same output port resulting from simulating the IC design with and without mutations. When output waveforms are different and yet the difference cannot be detected by the test program, there is likely a problem hidden in the test program. The severity of a hidden problem and the probability of the hidden problem being uncovered and subsequently fixed depend to a large degree on the nature and characteristics of the waveform difference.
For example, an IC design code and a set of test programs are received by a mutation-based test program evaluation system. By simulating or emulating the design code with the test programs, the mutation-based test program evaluation system obtains a set of waveforms from the output ports of the design code. The waveforms are called reference waveforms. Then, a small change, called a mutation or fault, is inserted into the design code, and the tests are re-run. The outputs are then called faulty outputs and the waveforms thus collected are called faulty waveforms. Depending on the mutation, some tests may generate faulty waveforms that are different from the reference waveforms, and others may generate faulty waveforms that are identical to the reference waveforms. Those runs producing different waveforms on output ports may or may not be detected by the test programs. The undetected differences may show hidden problems in the test programs. When there is a significant number of fault-test runs that produce undetected differences on faulty waveforms, users need guidance on which fault-test pair or pairs are more promising to look into.
Therefore, a method for ranking fault-test pairs in tracking down hidden problems and subsequently fixing the problems is desired.